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Abstract 

Strategic  planning  at  an  Air  Force  wing  is  a  team  based  activity  that  offers  many 
advantages  to  the  Air  Force.  But,  it  is  clear  there  are  still  questions  about  how  to  properly 
conduct  strategic  planning  to  produce  the  highest  quality  plans  in  the  most  effective  and 
efficient  manner.  To  answer  this  question,  the  Air  Force  created  a  research  project  aimed 
at  discovering  new  methods  to  improve  strategic  planning  at  Air  Force  wing  levels  using 
a  computer-mediated  communication  system.  The  research  project’s  goal  is  to  apply 
collaborative  processes  and  technologies,  such  as  Group  Support  Systems  (GSS),  to 
improve  the  quality  of  the  strategic  planning  process.  This  thesis  validated  the 
measurement  instruments  and  analyzed  the  data  from  one  of  those  studies.  The  thorough 
evaluation  of  the  measurement  instruments  led  to  validated  scales  that  could  be  valuable 
for  future  use.  The  analysis  shows  a  significant  increase  in  overall  plan  quality,  and 
increased  participant  satisfaction  vvdth  both  the  process  employed  and  the  product  created 
when  the  groups  used  a  GSS  instead  of  traditional  face-to-face  methods.  These  results 
lend  support  to  the  continued  fielding  of  such  systems  to  support  the  decision  making 
process  for  these  types  of  problems. 
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EFFECTS  OF  GROUP  SUPPORT  SYSTEMS  ON  UNITED  STATES 


AIR  FORCE  STRATEGIC  PLANNING  EFFORTS 

I.  Introduction 

Background 

Trying  to  make  the  federal  government  more  efficient  and  effective  is  oxymoronic 
to  some,  but  the  Government  Performance  Review  Act  (GPRA)  passed  in  1993  requires 
agencies  to  better  measure  their  performance  in  terms  of  results  rather  than  dollars  spent. 
However,  even  getting  the  agencies  to  comply  with  the  first  phase  of  the  GPRA,  which 
first  requires  them  to  develop  acceptable  strategic  plans,  has  become  a  monumental  task 
(Lovelace  and  Young,  1995).  Under  the  GPRA,  federal  agencies  must  submit  five-year 
plans  to  Congress  and  the  White  House  that  include  specific  goals  and  objectives.  By 
law,  each  agency’s  five-year  plan  must  include:  a  mission  statement;  goals  and 
objectives;  specifics  on  how  goals  and  objectives  will  be  achieved;  an  explanation  of  how 
performance  goals  relate  to  general  goals;  factors  that  could  affect  achievement  of  goals 
and  objectives;  a  description  of  evaluations  used  in  setting  goals  and  a  schedule  for  future 
evaluations. 

Strategic  Planning  in  the  Department  of  Defense 

Strategic  planning  in  the  post-Cold  War  era  has  proven  to  be  exceptionally 
problematic.  The  multitude  of  constantly  changing  national  and  international  tensions  that 
were  generally  ignored  because  they  were  overshadowed  by  the  east-west  confrontation 
of  the  Cold  War  now  combine  to  create  a  world  filled  with  diverse  challenges  to  U.S. 
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interests  (Managing  Change,  1992).  Equally  disturbing  is  the  fact  that  these  challenges 
are  not  as  clearly  defined  and  easily  articulated  as  the  monolithic  Soviet  threat  (Lovelace 
and  Young,  1995).  The  inherent  stability  the  Cold  War  provided  in  U.S.  strategic 
planning  is  now  missing.  This  missing  stability  changes  the  old  list  of  basic  elements 
needed  in  a  strategic  military  plan  to  a  new  list  headed  with  a  need  for  flexibility  and 
closely  followed  by  integration  of  action  plans  (Lovelace  and  Young,  1995). 

United  States  Air  Force  Strategic  Planning 

Strategic  planning  is  performed  throughout  the  U.S.  Air  Force.  It  is  used  as  a  core 
leadership  decision  making  process  to  determine  the  focus  and  direction  of  all  component 
wings.  The  strategic  planning  process  is  a  vital  element  in  the  distribution  and  allocation 
of  scarce  resources.  Each  wing  in  the  U.S.  Air  Force’s  Air  Combat  Command  is  required 
to  produce  an  annual  strategic  plan  (Murdock,  1997).  The  cxuxent  process  is  extremely 
time  consuming,  taking  three  to  four  months  to  complete  at  most  wings  (Adkins, 
Nimamaker,  Shearer,  Simcox,  and  Romero,  1998). 

Strategic  planning  at  an  Air  Force  wing  is  a  team  based  activity  that  is  critical  to 
mission  success.  Using  a  squadron  team  to  develop  a  strategic  plan  offers  many 
advantages  to  the  Air  Force.  The  team  based  process  allows  for  sharing  of  critical 
information,  generating  ideas,  making  decisions,  reviewing  the  effects  of  the  decisions, 
and  developing  commitment  to  the  strategic  plan  produced  (Murdock,  1997).  These 
decision  making  groups,  though,  are  social  entities  which  require  the  effective 
coordination  of  time  and  resources.  Their  goal  is  to  determine  the  optimal  solution  to  the 
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presented  issues  which  confront  them  within  the  time  frame  each  member  can  contribute 
(Adkins,  Nunamaker,  Shearer,  Simcox,  and  Romero,  1998). 

Problem  Statement 

The  main  challenge  for  most  senior  wing  leadership  is  how  to  produce  a  product 
that  is  both  meaningful  and  useful  for  the  wing.  Commitment  to  an  imusable  product  is  a 
hopeless  endeavor,  insupportable  by  senior  leadership.  A  second  important  challenge  is 
how  to  accomplish  this  effort  in  a  reasonable  amount  of  time.  Air  Force  strategic 
planning  teams  need  to  meet  and  engage  in  focused  planning  efforts  which  effectively 
coordinate  time  and  resources  to  produce  an  optimal  solution. 

Research  Focus 

It  is  clear  that  there  are  still  questions  about  how  to  properly  go  about  strategic 
planning.  Oft  asked  questions  about  strategic  planning  include  how  to  go  about  it  in  the 
most  effective  and  efficient  manner.  To  answer  this  question,  the  Air  Force  has 
commissioned  several  research  studies.  One  such  project  was  created  to  research 
methods  of  improving  the  Air  Force’s  strategic  planning  at  Air  Force  wing  levels  using  a 
computer-mediated  communication  system  as  a  tool.  The  research  project’s  goal  is  to 
apply  collaborative  processes  and  technologies  to  improve  the  quality  of  the  strategic 
planning  process  for  the  Air  Force.  The  current  phase  is  designed  to  produce  a  computer- 
mediated  strategic  planning  process  which  optimizes  the  application  of  group  support 
system  technology  and  group  facilitation.  The  group  support  system  in  use  is  a  suite  of 
team-based  decision  software  tools  that  shorten  the  cycle  time  for  strategic  planning, 
problem  solving,  and  other  processes,  such  as  voting  and  prioritizing  alternatives.  This 
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thesis  will  evaluate  the  efforts  of  the  Air  Force  research  project  to  improve  strategic 
planning  at  Air  Force  wing  level  using  a  computer-mediated  communication  group 
support  system. 
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II.  Literature  Review 


Introduction 

According  to  a  Business  Week  special  edition,  “change  can  [no  longer]  be  an 
occasional  episode  in  the  life  of  a  corporation.  Companies  with  rigid  structures  will  be 
swept  away.  Corporate  cultures  that  can  adapt  will  survive  and  thrive...”  (Managing 
Change,  1992:62).  Flexibility — ^as  well  as  the  related  constructs  of  speed,  adaptability, 
and  change — ^has  been  touted  as  a  key  tenet  of  the  “paradigm  for  the  postmodern 
manager”  (Byrne,  1992:62).  To  yield  positive  results,  change  and  flexibility  necessitate 
prior  effective  strategic  planning  (Mintzberg,  1997:25).  However,  researchers  have  not 
yet  conclusively  determined  why  some  planning  efforts  are  more  successful  than  others  in 
meeting  the  challenge  to  plan  strategically. 

Strategy 

Strategy  is  the  term  used  to  describe  how  a  business  will  accomplish  its  mission. 

It  is  the  path  selected  to  stay  competitive  in  a  market.  A  strategy  is  needed  when  the 
potential  actions  or  responses  of  intelligent  opponents  can  seriously  affect  the  endeavor’s 
outcome  (Mintzberg  and  Quinn,  1983:6). 

The  well-known  management  strategist  Henry  Mintzberg  has  proposed  four 
reasons  why  organizations  need  strategies.  They  are  to: 

1.  Set  direction.  Organizations  need  strategy  to  set  direction  for  themselves  and 
to  outsmart  competitors,  or  at  least  enable  themselves  to  maneuver  through  threatening 
environments  (Mintzberg,  1987:25).  Organizations  set  their  pace  based  on  their 
capabilities  and  the  actions  of  competitors. 
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2.  Focus  Effort.  Strategy  is  needed  to  focus  effort  and  promote  coordination  of 
activity  (Mintzberg,  1987:26).  Strategy  provides  the  common  purpose  for  all  members  of 
the  organization  so  they  can  work  together  towards  that  same  purpose. 

3.  Define  the  Organization.  Strategy  serves  not  only  to  direct  the  attention  of  the 
members  of  the  organization,  but  also  to  give  the  organization  meaning  for  them  as  well 
as  outsiders  (Mintzberg,  1987:27).  In  this  sense,  strategy  defines  the  organization’s 
reason  for  being.  It  tells  customers,  shareholders,  and  employees  why  the  firm  is  in 
business  and  what  the  firm  can  do  for  them. 

4.  Provide  Consistency.  Strategy  reduces  uncertainty,  provides  consistency,  and 
promotes  efficiency  (Mintzberg,  1987:29).  Strategy  gives  structure  to  the  organization 
and  outlines  the  function  of  the  firm  and  its  employees. 

An  organization  establishes  its  strategy  to  capitalize  on  its  strengths  and  the 
weaknesses  of  its  opponents.  This  strategy  provides  the  stability  needed  for  an 
organization  to  survive  and  thrive  in  its  environment. 

Strategic  Planning 

Strategic  planning  is  a  process  which  organizations  use  to  prepare  for  the  future 
and  is  the  foundation  for  establishing  the  organization’s  competitive  strategy.  It  is  the 
plan  the  organization  follows  to  keep  or  gain  a  competitive  edge  in  their  environment. 
Strategic  planning  involves  the  analysis  of  the  environment,  its  present  competitive 
strategy,  and  the  establishment  of  the  vision  and  mission  statements  which  define  the 
organization’s  quest  for  success. 


6 


Peter  Drucker  calls  strategic  planing  a  continuous  process  of  making  present 
entrepreneurial  (risk-taking)  decisions  systematically  with  the  greatest  knowledge  of  their 
futurity,  organizing  systematically  the  efforts  needed  to  carry  out  these  decisions,  and 
measuring  the  results  of  these  decisions  against  the  expectations  through  organized, 
systematic  feedback  (Drucker,  1973:125). 

This  planning  is  constantly  updated  to  reflect  the  organization’s  ever  changing 
environment,  it’s  competitive  processes,  and  risk  taking.  At  times,  an  organization  must 
take  risks  to  try  new  methods.  Presently,  strategic  planning  is  no  longer  a  luxury  -  it’s  a 
necessity  for  organizations  that  want  to  survive  and  prosper  (French,  1993:37-38). 
Strategic  Planning  in  Government 

Strategic  planning  in  government  agencies  is  slightly  different  than  in  private 
industries.  Government  agencies  are  affected  by  periodic  leadership  turnover,  separation 
of  power  and  checks  and  balances,  and  the  lack  of  a  profit  motive  (Moskow,  1978:27-3 1). 
Leadership  can  change  every  four  years  with  the  election  of  a  new  president  who  may 
have  different  goals  and  objectives  than  the  current  administration.  A  plan  developed 
during  one  administration  may  not  survive  to  be  realized  in  the  successive.  Moskow 
states  that  in  the  corporate  world,  managers  have  a  much  longer  relationship  with  an 
organization,  and  often  the  present  Chief  Executive  Officer  chooses  and  grooms  the 
successor  (Moskow,  1978:32). 

The  Constitution  of  the  United  States  created  a  set  of  checks  and  balances  which 
does  not  allow  the  President  to  solely  enact  his  own  policies.  Instead  all  policies  must  be 
approved  by  Congress  and,  when  challenged,  be  judged  legal  by  the  court  systems.  Plans 
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must  continually  be  reviewed  to  reflect  these  realities.  Congressional  disapproval  and 
critical  public  opinion  can  end  a  program  before  any  real  progress  can  be  made. 

Finally,  the  major  difference  between  plaiming  in  government  agencies  and 
private  enterprises  is  the  lack  of  a  governmental  profit  motive.  A  program  can  be  judged 
a  success  or  failure  purely  based  on  its  acceptance  by  the  public  or  congressional 
advocates.  Though  there  are  difficulties  in  the  implementation  of  strategic  planning  in 
government,  “better  planning  leads  to  more  rational  decision  making  and  better 
government  policies  and  programs”  (Moskow,  1978:58). 

The  Government  Performance  and  Results  Act.  Public  Law  No  103-62,  the 
Government  Performance  and  Results  Act  of  1993,  was  enacted  to  “improve  the 
efficiency  and  effectiveness  of  Federal  programs  by  establishing  a  system  to  set  goals  for 
program  performance  and  to  measure  results”  (Senate  Report  No  103-58, 1993:328).  The 
law  established  requirements  for  Federal  agencies  to  develop  strategic  plans,  performance 
plans,  and  performance  measures.  According  to  the  act,  strategic  plans  must  include  a 
detailed  mission  statement,  and  present  the  long-term  goals  and  resources  needed  to  meet 
the  goals.  The  performance  plans  must  show  what  annual  action  plans  need  to  be 
accomplished  at  each  level  in  order  for  the  higher  level  to  meet  its  own  goals.  The 
strategic  plans  should  also  describe  the  measures  and  means  to  verify  and  validate  their 
measures  (Senate  Report  No  103-58, 1993:341).  Finally,  the  annual  performance  reports 
will  review  the  progress  of  the  agency  in  meeting  its  goals.  These  reports  will  provide 
the  feedback  to  all  managers,  policy  makers,  and  the  public  on  what  was  actually 
accomplished  over  the  past  year  (Senate  Report  No  103-58, 1993:342). 
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Department  of  Defense  Strategic  Planning 


Strategic  planning  is  a  challenging,  but  necessary,  endeavor  for  any  organization, 
small  or  large.  For  the  U.S.  Department  of  Defense  (DoD)  it  is  a  solemn  responsibility  to 
the  Nation.  President  Eisenhower  said  in  1958,  “No  military  task  is  of  greater  importance 
than  the  development  of  strategic  plans  which  relate  our  revolutionary  new  weapons  and 
force  deployments  to  national  security  objectives”  (Eisenhower,  1958).  In  spite  of  its 
attention  to  strategic  planning,  DoD  has  not  enjoyed  great  success  in  this  area.  For 
example,  a  congressional  staff  report  characterized  DoD’s  strategic  planning  as  haphazard 
at  best.  It  specifically  stated  that  “Inattention  to  strategic  planning  has  led  to  numerous 
deficiencies,  including  a  lack  of  clarity  of  DoD’s  strategic  goals”  (Senate  Staff  Report, 
1985).  They  concluded  that  the  stated  goals  are  vague  and  ambiguous.  In  an  organization 
as  large  as  DoD,  the  clear  articulation  of  overall  strategic  goals  can  play  an  important  role 
in  achieving  a  coordinated  effort  toward  these  goals  by  the  various  components  and 
individuals  within  them.  Clarity  of  goals  can  enhance  unity  and  integration.  DoD  loses 
the  benefit  of  this  unifying  mechanism  through  its  failure  to  clarify  its  strategic  goals.  To 
correct  this  problem  and  other  strategic  planning  deficiencies,  DoD  needs  to  establish  and 
maintain  a  well-designed  and  highly  interactive  strategic  planning  process”  (Senate  Staff 
Report,  1985). 

Follovidng  up  on  this  staff  finding,  Congress,  in  the  Goldwater-Nichols 
Department  of  Defense  Reorgani2ation  Act  of  1986  (GNA),  mandated  a  hierarchical 
process  for  strategic  direction,  strategic  planning,  and  contingency  planning  for  the  U.S. 
Armed  Forces  (Goldwater-Nichols  Act,  1986).  This  process  was  designed  to  improve 
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strategic  planning  by  harmonizing  strategic  direction  and  planning  with  the  development 
of  defense  programs  that  would  enable  DoD  to  achieve  its  strategic  goals.  It  was  also 
designed  to  integrate  the  strategic  and  operational  planning  conducted  by  the  combatant 
Commanders-in-Chiefs  (CINCs).  To  these  ends,  the  Chairman  of  the  Joint  Chiefs  of 
Staff  (CJCS)  was  assigned  key  and  specific  responsibilities.  Since  passage  of  the  GNA, 
which  is  now  codified  in  Title  10,  United  States  Code  (10  USC),  the  JCS  and  then  the 
Chairman  have  developed,  implemented,  and  revised  specific  processes  for  fulfilling 
most  of  these  statutory  responsibilities. 

Enactment  of  the  GNA  notwithstanding,  a  recent  study  concluded  that  the 
strategic  planning  currently  conducted  by  the  Joint  Staff,  on  behalf  of  the  Chairman,  does 
not  adequately  establish  and  specify  strategic  objectives  nor  does  it  integrate  and  establish 
priorities  for  them.  In  short,  current  strategic  planning  for  the  U.S.  Armed  Forces  is  of 
limited  use  in  planning  for  future  military  capabilities  and  integrating  the  planning 
conducted  by  the  CINCs.  It  should  ensure  that  both  of  these  efforts  conform  to  national 
military  and  security  objectives.  Equally  disturbing,  it  does  not  provide  sufficient 
underlying  rationale  for  the  review  of  service  functions  nor  does  it  provide  unequivocal 
and  compelling  bases  for  the  development  and  implementation  of  joint  doctrine 
(Lovelace  and  Young,  1995). 

Strategic  Plaiming  and  Performance 

Steiner  (1972)  argued  for  the  importance  of  strategic  planning,  providing  insight 
into  overcoming  the  barriers  and  biases  associated  with  planning  failures.  However, 
research  by  Steiner  and  others  is  founded  in  the  critical  assumption  that  planning  is 
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important.  The  literature  is  inundated  with  the  apparent  advantages  of  planning,  most 
notably  its  ability  to  improve  the  fit  between  the  organization  and  its  external 
environment  (Godiwalla,  Meinhart,  and  Warde,  1981).  Others  have  argued  that  planning 
aids  in  the  identification  of  future  marketing  threats  and  opportunities,  elicits  an  objective 
view  of  managerial  problems,  creates  a  firamework  for  internal  communication,  promotes 
forward  thinking,  and  encourages  a  favorable  attitude  to  change  (Loasby,  1976;  Stem, 
1966;  Wilson,  1973).  Further,  there  are  intrinsic  benefits  that  accme  as  a  result  of  the 
planning  process  (Greenley,  1986). 

Langley  (1988)  also  provided  support  for  the  benefits  of  planning,  identifying 
four  roles  of  formal  strategic  planning.  In  the  public  relations  role,  formal  strategic 
planning  is  intended  to  impress  or  influence  outsiders.  The  information  role  provides 
input  for  management  decisions.  The  group  therapy  role  is  intended  to  increase 
organizational  commitment  through  the  involvement  of  people  at  all  levels  of  the 
organization  in  strategic  planning.  Finally,  the  direction  and  control  role  is  fulfilled  when 
plans  serve  to  guide  future  decisions  and  activities  toward  some  consistent  ends. 

According  to  Roach  and  Allen  (1983),  the  strategic  planning  process  is  the 
product  of  the  best  minds  inside  and  outside  the  corporation.  The  process  considers 
future  implications  of  current  decisions,  adjusts  plans  to  the  emerging  enviromnent, 
manages  the  business  processes  analytically,  and  links,  directs,  and  controls  complex 
enterprises  through  a  practical,  working  management  system.  This  process  plays  a  vital 
role  in  firm  performance  (Roach  and  Allen,  1983). 
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Cartwright  (1987)  suggested  that  effective  planning  is  not  as  rational  and 
analytical  as  it  has  been  portrayed  in  the  literature.  He  argues  for  the  lost  art  (rather  than 
science)  of  planning.  He  contends  that  planning  is  both  a  generic  activity  whose  success 
determinants  are  partially  independent  of  the  area  in  which  it  is  applied,  and  an  area 
where  judgment,  intuition,  and  creativity  are  still  important. 

Schwenk  and  Shrader  (1993)  conducted  a  meta-analysis  of  fourteen  studies  on 
formal  strategic  planning  and  performance.  They  concluded  that  strategic  planning 
promotes  long-range  thinking,  reduces  the  focus  on  operational  details,  and  provides  a 
structured  means  for  identifying  and  evaluating  strategic  alternatives.  Since  this  was  the 
first  review  that  clearly  demonstrated  the  planning-performance  link  across  studies,  it 
strengthened  the  case  for  recommending  the  use  of  strategic  planning  in  all  firms, 
regardless  of  size. 

Sinha  (1990)  empirically  established  some  kind  of  a  planning-performance 
linkage.  Sinha  examined  1087  decisions  made  by  129  Fortune  500  firms  between  1982 
and  1986.  He  concluded  that  characteristics  of  the  decisions  accoimted  for  15  percent  of 
the  variance  in  data  and  therefore  should  be  regarded  as  important  determinants  of  the 
contribution  planning  makes  to  decision  making.  However,  Sinha  concedes  that  the 
quality  of  planning  is  critical  to  the  relationship  (1990:487-491). 

Computer-mediated  Strategic  Planning 

A  computer-mediated  strategic  planning  process  is  supposed  to  help  reduce  the 
constraints  associated  with  bringing  a  large  group  of  people  together  to  collaborate. 
Specifically,  Group  Support  Systems  (GSS)  are  technology  designed  to  directly  impact 
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and  change  the  behavior  of  groups  to  improve  group  effectiveness,  efficiency  and 
satisfaction  (Nunamaker,  Dennis,  Valacich,  Vogel,  and  George,  1991).  GSSs  have  been 
designed  to  reduce  the  effects  of  the  barriers  to  ideal  group  decision  making  (Briggs, 
1998).  According  to  Valacich,  Dennis,  and  Nunamaker  (1992),  "a  group  support  system 
(GSS),  is  described  as  an  environment  that  contains  a  series  of  networked  computer 
workstations  that  enable  groups  to  meet  face-to-face,  with  a  computer-supported 
electronic  communication  channel  used  to  support  or  replace  verbal  communication" 
(1992: 49-50).  When  a  GSS  is  applied  to  group  decision  making,  ideas  can  be  exchanged 
between  group  members  and  organized  into  distinct  categories.  These  categories  can  then 
be  analyzed  by  group  members  exchanging  information  through  electronic  file  folders. 
Consensus  can  be  developed  between  group  members  utilizing  the  GSS,  and  the  data  can 
be  used  and  reviewed  in  future  meetings  or  exported  to  a  superior  or  expert  for  critique  or 
approval. 

Group  Support  Systems  are  in  wide  use  today  and  are  likely  to  be  even  more 
widely  used  in  the  future  to  allow  geographically  separated  people  to  work  together 
through  computer  networks.  GSSs  are  being  used  with  increasing  frequency  in  a  variety 
of  organizations  (Brashers,  Adkins,  and  Meyers,  1994;  Nunamaker,  Briggs,  Mittleman, 
Vogel,  and  Balthazard,  1997;  Vogel,  Martz,  Nunamaker,  Grohowski,  and  McGoff,  1990) 
where  computer  technology  has  been  a  vehicle  for  change.  A  GSS  can  have  critical 
impacts  on  group  interaction  especially  in  overcoming  barriers  that  influence  group 
decision  making.  For  example,  research  has  shown  that  in  a  face-to-face  group  meeting, 
20%  of  the  people  do  80%  of  the  talking  because  some  group  members  are  shy,  of  lesser 
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status,  intimidated,  or  too  polite  (Kirkpatrick,  1992).  This  lack  of  participation  among 
group  members  may  lead  to  lower  overall  productivity,  or  less  critical  evaluation  of  ideas. 
In  an  environment  using  a  GSS,  anonymous  computer-mediated  communication  provides 
the  opportunity  for  all  group  members  to  participate  equally  (Nunamaker,  Dennis, 
Valacich,  Vogel,  and  George,  1991).  Equal  participation  has  the  potential  to  improve  the 
quality  of  interaction  and  perhaps  to  provide  opportunity  for  more  critical  discussion  of 
decision  alternatives  (Brashers,  Adkins,  and  Meyers,  1994;  Jessup  and  Valacich,  1993). 
Thus,  people  who  have  used  GSSs  claim  that  these  meetings  are  more  effective  than  face- 
to-face  interactions  for  group  planning,  problem  solving,  decision  making,  and  group 
interaction  (Dennis,  Heminger,  Nunamaker,  and  Vogel,  1990;  Vogel,  Martz,  Nunamaker, 
Grohowski,  and  McGoff,  1990). 

Effectiveness  of  GSS 

GSS  research  has  looked  at  several  kinds  of  variables,  including  measures  of 
effectiveness  and  efficiency.  The  first  dimension  reflects  a  goal-centered  approach  to 
assessing  organizational  effectiveness.  The  goal  attainment  measure  is  primarily 
concerned  with  the  specific  end  results  normally  anticipated  from  a  planning  session.  The 
second  dimension  follows  the  tradition  of  earlier  studies  that  sought  to  examine  the 
impact  of  GSS  use  on  time  to  complete  complicated  planning  tasks. 

Attempts  to  understand  the  group  processes  that  occur  during  computer  supported 
sessions  have  relied  on  previous  research  of  face-to-face  groups.  During  a  GSS  session, 
each  member  can  act  as  a  lone  problem  solver  and  exert  efforts  on  only  one  aspect  of  the 
problem.  To  incorporate  this  ability  into  the  previous  research  requires  studying  both  the 
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individual  and  the  group  problem  solving  process  and  how  they  are  coordinated  by  the 
GSS  (Turoff,  Hiltz,  Bahgat,  and  Rana,  1993). 

While  GSS  implementations  are  on-going,  understanding  of  their  proper  usage  is 
still  not  clear.  GSSs  have  been  applied  to  many  different  types  of  problems  with  mixed 
results.  While  several  studies  quote  current  periodicals  as  claiming  dramatic 
improvements  by  using  a  GSS,  no  study  claimed  to  have  found  previous  research  with 
such  conclusions,  nor  reported  any  dramatic  results  of  their  own.  It  should  be  noted  that 
even  though  sales  of  GSS  software  are  increasing  and  many  corporations  are  rushing  to 
implement  them,  much  of  the  empirical  evidence  is  confined  to  laboratory  experimental 
conditions  and  case  studies,  with  few  field  studies  available  at  this  time.  Overall, 
however,  researchers  agree  that  meetings  requiring  decision-making  are  more  effective 
when  employing  a  GSS  (Adkins,  Nunamaker,  Shearer,  Simcox,  and  Romero,  1997; 
Jarvenpaa  Rao,  and  Huber,  1988:649;  Lam,  1997:194,  Vogel,  Martz,  Nimamaker, 
Grohowski  and  McGoff,  1990:42).  The  measure  of  effectiveness,  though,  varies  with  the 
research. 

GSS  and  Quality 

One  of  the  most  widely  researched  measures  of  GSS  effectiveness  has  been  the 
quality  of  the  decision  made.  For  example,  in  a  study  applying  a  GSS  to  problem  finding 
tasks,  group  decision  quality  was  found  to  be  statistically  higher  in  GSS  supported  groups 
(Gallupe  and  DeSanctis,  1988).  The  experimental  task  was  to  find  the  underlying 
problem  in  a  crisis  situation.  By  manipulating  the  task  difficulty,  the  authors  attempted  to 
determine  how  a  GSS  would  affect  group  decision  making  for  problem  finding  tasks  of 
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varying  difficulty.  They  reported  that  the  use  of  a  GSS  increased  the  number  of 
alternatives  considered  and  improved  the  decision  quality  for  problem  solving  tasks, 
especially  in  the  higher  difficulty  task.  Dennis  and  others  (1990)  also  performed  a 
comprehensive  review  of  GSS  literature  and  found  that  several  studies  show  that  GSS  use 
leads  to  better  quality  decisions  (Dennis,  George,  Jessup,  Nunamaker,  and  Vogel, 
1988:602).  These  and  other  studies  have  supported  the  finding  that  GSS  use  allows 
organizations  to  develop  better,  higher  quality  solutions  to  problems  and  make  higher 
quality  decisions  (Cullen,  1997:49;  Dennis,  Heminger,  Nunamaker,  Vogel,  1990:113; 
Jarvenpaa,  Rao,  and  Huber,  1988:658;  Lam,  1997:204;  Nunamaker,  Briggs,  Mittleman, 
Vogel,  and  Balthazard,  1996:171).  These  results  led  to  the  formation  of  Proposition  1. 

Proposition  1:  Higher  quality  action  plans  will  be  produced  by  groups  employing 
GSS  than  by  traditional  Face-to-Face  groups. 

GSS  and  Satisfaction 

Another  commonly  used  research  measure  of  GSS  use  has  been  satisfaction. 

While  the  results  of  these  studies  has  been  mixed,  especially  early  in  the  history  of  GSS 
research,  a  recent  meta-analysis  of  29  studies  comparing  GSS  supported  versus  non¬ 
supported  group  work,  revealed  that  GSS  use  improves  satisfaction  with  the  meeting 
process  and  confidence  in  the  decision  reached  (Anson,  Bostrum,  and  Wynne,  1995:190- 
191).  A  possible  explanation  of  the  mixed  results  is  the  prevalence  of  laboratory  studies 
early  in  GSS  research  efforts  (Nunamaker,  Briggs,  Mittleman,  Vogel,  and  Balthazard, 

1 997).  The  use  of  a  GSS  in  organizations  was  producing  positive  effects  not  modeled  in 
the  lab  because  real  groups  do  not  perform  in  a  void,  but  rather  in  an  organizational 


context.  This  context  drives  the  attitudes  and  behaviors  in  group  meetings  (Nunamaker, 
Briggs,  Mittleman,  Vogel,  and  Balthazard,  1997).  Much  of  the  work  that  followed  used 
field  studies  which,  in  contrast  to  the  earlier  work,  reported  positive  reactions  and  results 
from  GSS  use  (Dennis  and  Gallupe,  1993). 

But,  even  in  these  studies  the  use  of  the  satisfaction  measure  is  not  consistent. 
Several  studies  measure  the  satisfaction  with  the  decision  making  process  while  others 
measure  the  satisfaction  or  commitment  to  the  decision  reached  (Anson,  Bostrum,  and 
Wynne,  1995;  Nunamaker,  Briggs,  Mittleman,  Vogel,  and  Balthazard,  1997).  Only 
recently  have  efforts  been  made  to  separate  these  measxires  and  study  the  performance  of 
GSS  supported  groups  in  a  more  rigorous  fashion  (Dennis  and  Gallupe,  1993:64). 

One  possible  explanation  for  the  increase  in  satisfaction  is  that  the  ability  for 
meeting  participants  to  work  and  communicate  in  parallel  prohibits  any  one  member  of 
the  group  from  dominating  or  controlling  the  group’s  discussions  (Tyran,  Dennis,  Vogel, 
and  Nunamaker,  1 992:3 1 7).  Since  each  member  has  equal  access  to  the  communications 
channels,  domination  is  eliminated  and  free  expression  is  available  (Pinsonneault  and 
Heppel,  1997:89).  Another  possibility  is  that  the  anonymity  of  the  inputs  prevents  higher 
ranking  members  from  dominating  the  meeting  while  allowing  shy  members  to  remain 
separate  from  their  statements  (Pinsonneault  and  Heppel,  1997:89).  The  differences  in 
power,  position  or  status  are  equalized  due  to  anonymity  leading  to  a  reduction  in 
personality  conflicts  and  an  increase  in  satisfaction  with  the  meeting  process  (Talbott, 

1 995 : 1 1 7).  As  stated  before,  this  equality  of  participation  has  been  found  to  improve  the 
quality  of  interaction  (Brashers,  Adkins,  and  Meyers,  1994;  Jessup  and  Valacich,  1993). 
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Valacich,  1993).  Thus,  people  who  have  used  GSS  claim  that  these  meetings  are  more 
satisfying  than  face-to-face  for  group  interaction  (Dennis,  Heminger,  Nunamaker,  and 
Vogel,  1990;  Vogel,  Martz,  Nunamaker,  Grohowski,  and  McGofif,  1990).  This  type  of 
increase  in  satisfaction  would  seem  most  readily  identified  as  satisfaction  with  the 
process  of  decision  making  and  is  addressed  as  such  here.  These  results  led  to  the 
formation  of  Proposition  2. 

Proposition  2:  Groups  employing  GSS  will  report  a  higher  satisfaction  with  the 
decision  process  than  traditional  Face-to-Face  groups. 

In  another  study  researching  the  effects  of  various  computer  supports  for  groups 
working  on  xmstructured  problems,  the  GSS  groups  produced  mixed  results  for  quality, 
but  higher  satisfaction  (Jarvenpaa,  Rao,  and  Huber,  1988).  Groups  of  software  engineers 
were  asked  to  work  on  a  software  design  problem.  The  main  focus  of  the  exercise  was 
the  gathering  of  ideas  and  the  reaching  of  consensus.  Some  groups  experienced  increased 
idea  generation,  more  thorough  discussions  of  the  ideas,  and  better  quality  performance 
than  their  traditional  counterparts,  while  other  teams  did  not  (Jarvenpaa,  Rao,  and  Huber, 
1988).  But,  noteworthy  in  this  article  is  the  result  that  the  GSS  supported  members 
reported  a  higher  satisfaction  with  their  solutions  than  did  their  non-supported 
counterparts.  This  type  of  increase  in  satisfaction  would  seem  most  readily  identified  as 
satisfaction  with  the  decision  reached  and  is  addressed  as  such  here. 

These  results  lead  to  the  formation  of  Proposition  3. 

Proposition  3:  Groups  employing  GSS  will  report  a  higher  satisfaction  with  the 
action  plan  produced  than  traditional  Face-to-Face  groups. 
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GSS  and  Efficiency 


A  widely  used  research  variable  in  GSS  studies  has  been  efficiency.  The 
interaction  of  task  type  and  technology  on  group  performance  and  member  reactions  was 
examined  by  Straus  and  McGrath.  They  examined  the  impact  computer-mediated  and 
face-to-face  communications  have  on  group  performance.  They  investigated  the 
differences  between  computer-mediated  and  face-to-face  groups  when  performing  idea- 
generation,  intellective,  and  judgment  tasks.  They  argued  that  “different  tasks  vary  in 
how  much  social  context  information  their  effective  execution  requires”  (Straus  and 
McGrath,  1994:  88).  The  authors  found  from  their  experiments  that,  in  general,  when 
looking  at  judgment  tasks,  face-to-face  groups  will  be  more  productive  and  superior  in 
performance  when  compared  to  computer-mediated  groups  (Straus  and  McGrath,  1994: 
94).  However,  the  authors  note  that  computer-mediated  groups  are  more  appropriate  for 
intellective  and  idea-generation  tasks,  particularly  when  productivity  is  a  priority  or  time 
is  at  a  premium  (Straus  and  McGrath,  1994:  95).  Before  closing  their  article,  the  authors 
reiterate  what  several  previous  studies  have  concluded,  stating  “it  also  is  important  to 
recognize  that  group  work  may  involve  multiple  processes.... The  results  of  this  study 
imply  that  different  media  are  more  or  less  appropriate  to  the  different  phases  of  a  group 
project”  (Straus  and  McGrath,  1994:  95).  Therefore,  depending  on  the  need  for  social 
context  cues,  face-to-face  communication  may  or  may  not  be  more  appropriate. 

As  found  in  several  of  the  studies  reported  above,  one  of  the  main  benefits  of  a 
GSS  is  that  it  generates  a  large  amovmt  of  specific  and  detailed  information  in  a  relatively 
short  amount  of  time.  The  GSS  allows  the  group  members  to  work  simultaneously  and 
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generate  parallel  input,  which  increases  the  speed  of  idea  generation  and  information 
sharing  (Briggs,  Nunamaker,  and  Sprague  1997:9).  Thus,  the  total  number  of  ideas 
generated  is  increased,  compared  to  conventional  meetings  (Aiken,  Vanjam,  and  Krosp, 
1995:39).  Furthermore,  a  GSS  incorporates  structure  into  the  decision-making  process 
(Dennis,  George,  Jessup,  Nunamaker,  and  Vogel,  1988:593;  Lam,  1997:194).  It  tends  to 
focus  the  group’s  problem-solving  efforts  and  process  (Briggs,  Nunamaker,  and  Sprague, 
1997:7).  Consequently,  a  GSS  enhances  overall  group  productivity  (Aiken,  Vanjani,  and 
Krosp,  1995:38). 

A  review  of  eight  case  studies  of  companies  using  various  levels  of  computer 
support  to  generate  strategic  plans  shows  similar  results  (Tyran,  Dennis,  Vogel,  and 
Nunamaker,  1992).  The  groups  studied  had  a  GSS  available,  but  were  not  compelled  to 
use  it.  In  general,  the  article  reports  that  the  best  results  came  when  the  groups  used  the 
GSS  to  generate  ideas,  solicit  managerial  input,  and  identify  key  issues.  But,  it  was  noted 
that  the  GSS  was  not  typically  used  to  resolve  differences.  Most  of  the  groups  studied 
followed  the  same  pattern  of  GSS  usage.  First,  they  would  use  the  GSS  to  collect  and 
organize  ideas  and  managerial  inputs,  then  the  group  would  disperse  into  departmental 
subgroups.  Finally,  they  would  reconvene  to  present  and  discuss  plans.  Post  process 
surveys  showed  that  the  members  felt  the  GSS  helped  their  group  generate  ideas,  identify 
the  key  ideas,  and  significantly  decrease  the  time  needed  to  come  to  a  decision  (Tyran, 
Dennis,  Vogel,  and  Nunamaker,  1992). 

These  results  lead  to  the  formation  of  Proposition  4. 
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Proposition  4:  Groups  employing  GSS  will  complete  the  action  plan  development 


process  faster  than  traditional  Face-to-Face  groups. 


21 


III.  Methodology 

Introduction 

To  investigate  how  humans  react  differently  between  computer  mediated  and 
face-to-face  decision  making  groups,  and  specifically  whether  a  GSS  would  support 
better  strategic  planning,  a  sponsor  interested  in  this  line  of  research  was  located  at 
Wright-Patterson  Air  Force  Base’s  Air  Force  Research  Laboratory,  AFRL/HESS.  The 
laboratory  already  had  a  number  of  research  projects  underway  in  this  area.  Upon 
meeting  with  the  sponsor,  an  existing  data  set  was  identified  that  needed  to  be  evaluated 
in  depth,  based  on  a  cursory  analysis  which  had  been  completed  previously  by  the 
University  of  Arizona.  There  were  indications  that  the  original  data  set  allowed  for 
additional  analysis  since  no  correlations  were  reported,  and  one  of  the  measurement 
instruments  purported  to  measure  two  different  constructs  was  reported  with  only  one 
alpha  level.  We  both  felt  the  previous  report  could  be  improved  with  a  more  in-depth 
background  literature  review  to  lend  stronger  theoretical  support  for  the  project’s 
approach.  Additionally,  further  statistical  analysis  of  the  collected  data  was  both  possible 
and  necessary  to  better  understand  the  impact  of  the  study.  The  data  set  came  from  a 
research  project  case  study  in  strategic  planning  done  at  the  366th  Wing,  Mountain  Home 
Air  Force  Base,  Idaho.  At  the  366th  Wing  there  is  a  Wing  Command  and  five  group 
level  units:  Wing  Staff  (WG),  Operations  Group  (OG),  Logistics  Group  (LG),  Support 
Group  (SPTG),  and  Medical  Group  (MDG).  Each  group  is  responsible  for  a  number  of 
squadrons  and  there  are  a  total  of  24  squadrons  in  the  366th  Wing. 
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Original  Procedure 


A  facilitated  methodology  for  the  strategic  planning  effort  was  established  using 
the  senior  leadership  at  the  Air  Force’s  366th  Wing  at  Moimtain  Home  Air  Force  Base, 
Idaho.  A  facilitator  worked  with  several  strategic  planning  groups  on  the  base  including 
the  366th  Wing  command,  three  group  level  organizations  (OG,  LG,  SPTG),  the  366th 
Wing  staff,  and  seven  squadrons.  These  efforts  were  completed  using  computer- 
supported  strategic  planning  methods  and  a  group  support  system  called  Group  Support 
System  (GSS)  that  was  developed  at  the  University  of  Arizona.  Quality  questionnaires 
were  also  administered  to  all  of  the  participants  in  each  squadron.  Data  was  collected 
from  all  squadrons,  both  those  using  traditional,  face-to-face  methods  and  those  using 
computer-supported  strategic  planning  methods.  All  squadron  action  plans  were 
evaluated  by  a  panel  of  seven  Quality  Improvement  experts  selected  from  other  bases 
using  a  six  item  quality  scale. 

The  computer-supported  strategic  planning  session  started  with  a  discussion  of  the 
function  of  mission  and  vision  statements  followed  by  a  review  of  the  previous  year’s 
mission  and  vision  statements  in  parallel  with  the  other  participants.  Then  the  group 
turned  its  efforts  to  authoring  new  mission  and  vision  statements.  Finally,  nominal  group 
techniques  and  anonymous  voting  were  used  to  select  the  final  statements  for  the  Wing. 

After  these  initial  efforts,  the  Wing  Command  brainstormed  ideas  for  Wing  goals 
using  the  mission  and  vision  statements  developed  in  the  initial  efforts  as  a  guide. 
Anonymous  voting  on  the  goals  followed  by  refinement  of  the  highest  rated  goals 
resulted  in  a  final  list  of  goals,  which  were  then  used  as  the  basis  for  development  of 
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Wing  objectives.  The  Wing’s  goals  were  arranged  in  a  hierarchical  tree  structure  so  those 
objectives  that  supported  each  goal  could  be  generated  in  parallel. 

A  similar  process  was  used  at  the  Group  Level  for  their  strategic  planning 
sessions.  The  tree  structure  of  Wing  goals  and  objectives  was  used  by  the  Group  staff 
members  for  each  of  the  Groups  to  develop  targets  in  parallel.  These  targets  were  then 
refined  as  needed  before  the  group  moved  on  to  the  next  objective. 

The  Squadron  Level  action  planning  sessions  for  each  of  the  seven  squadrons 
utilizing  the  GSS  followed  a  systematic  process.  First,  the  participants  reviewed  the 
definition  and  fimction  of  an  action  plan.  This  was  followed  by  a  discussion  of  the  action 
plan  template.  The  group  then  proceeded  to  review  the  Wing  and  Group  goals, 
objectives,  and  targets,  which  led  to  the  development  potential  action  plans,  once  more  in 
parallel.  Finally,  the  generated  action  plans  were  compared  to  evaluative  criteria. 

The  definition  for  action  plans  was  a  link  between  day-to-day  work  place 
activities  and  the  vision,  mission,  goals,  and  objectives  of  the  Wing.  The  teams  were 
presented  the  criteria  that  the  action  plans  developed  should  be  simple  and  easy  to  apply 
while  still  meeting  the  needs  of  the  squadron.  The  action  plans  also  have  to  be  focused  on 
those  day-to-day  processes  that  can  be  measured,  analyzed,  and  improved.  The  ideal 
design  of  the  action  plans  are  such  that  they  were  implementable,  acceptable,  and 
attainable.  The  action  plan  template  used  included  a  description,  metric,  milestone, 
success  criteria,  responsible  authority,  resource  identification,  and  feedback  mechanism. 
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Sample 


There  were  226  participants  from  the  366th  Wing.  At  the  squadron  level  there 
were  139  participants  (Males  =  105,  Females  =  21,  No  Answer  =  13).  The  mean  age  of 
the  squadron  participants  was  35.4  with  a  range  of  21  to  56  years  of  age.  The  majority  of 
the  squadron  participants  had  participated  in  one  or  less  computer  supported  meetings 
(N=l  07).  Representatives  from  22  of  24  squadrons  participated  in  the  research  project. 
Seven  squadrons  (N=92)  used  computer-  supported  strategic  planning  methods  and  17 
squadrons  (N=47)  used  traditional  face-to-face  strategic  planning  methods.  A  panel  of 
seven  Quality  Improvement  officers  from  other  Air  Force  bases  was  used  to  review  each 
squadron's  action  plans.  The  reviewers  had  a  mean  of  20.1  years  of  service  in  the  Air 
Force  and  a  mean  of  2.96  years  working  on  strategic  planning. 

Dependent  Variables 

A  six  item  quality  questionnaire  (see  Appendix  A)  was  created  for  the  expert 
reviewers  to  evaluate  each  squadron’s  action  plans.  The  reviewers'  questionnaire  asked 
one  question  each  on  the  quality  of  the  action  plans,  achievability,  buy-in,  how  well  the 
action  plans  addressed  the  targets,  and  how  clearly  the  plans  would  be  measured.  A  26 
item  satisfaction  questionnaire  was  administered  to  measure  the  partieipant’s  satisfaction 
with  the  strategic  planning  process  and  commitment  to  the  strategic  plans  produced.  The 
questionnaire  for  the  Non-GSS  supported  groups  is  in  Appendix  B,  while  the  survey 
administered  to  the  GSS  participants  is  in  Appendix  C.  The  time  needed  to  complete 
each  squadron’s  action  plan  was  measvued  via  a  questionnaire  for  the  traditional  groups 
and  actual  measurement  for  those  who  were  GSS  supported. 
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Analysis  Procedure 


The  new  analysis  started  with  a  validation  of  the  evaluator  responses  looking  for 
bias  in  results.  Reviewers  showing  a  significant  bias  were  eliminated  from  further 
analysis.  After  examining  the  reviewers,  a  factor  analysis  to  validate  all  measurement 
scales  was  done,  followed  by  a  scale  reliability  calculation  to  determine  the  value  of  the 
scale.  All  factor  correlations  were  calculated  and  reported.  Following  the  factor  analysis 
of  the  reviewer  scale,  t-tests  were  used  to  statistically  determine  if  there  was  a  qualitative 
difference  between  the  strategic  action  plans  generated  by  the  GSS  groups  as  compared  to 
the  traditional  face-to-face  groups. 

Analysis  of  the  participant  scale  also  began  vvdth  a  factor  analysis  and  scale 
reliability  test.  All  factor  correlations  were  calculated  and  reported.  After  that,  t-tests 
were  performed  to  statistically  determine  if  there  was  a  difference  in  identified  factors 
between  the  two  groups,  GSS  and  Non-GSS  supported  groups.  After  determining  if  there 
were  differences  between  the  quality  of  product  created  by  the  two  groups,  those 
identified  factors  were  used  in  a  linear  regression  Avithin  each  group  to  see  if  they 
predicted  the  expert’s  quality  assessment. 
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IV.  Analysis 


Original  Analysis 

The  original  results  reported  that  the  squadrons  which  used  Group  Support 
Systems  (GSS)  to  develop  strategic  plans  developed  higher  quality  strategic  plans  than 
those  squadrons  that  did  not  use  the  GSS  (t(7)=3.47,  p<.05).  There  was  no  significant 
difference  in  the  commitment  to  implementation  variable  between  the  GSS  and  the  non- 
GSS  squadrons  reported  in  the  initial  analysis.  Furthermore,  the  original  report  stated 
that  the  squadrons  that  used  computer-supported  strategic  planning  were  more  satisfied 
with  the  strategic  planning  process  than  those  squadrons  that  used  traditional  strategic 
planning  (t(137)=-2.28,  p<.05).  And  finally,  the  original  authors  reported  that  the 
strategic  planning  process  took  an  average  of  1 7.7  hours  for  squadrons  that  did  not  use 
the  GSS,  but  less  than  8  hours  for  those  squadrons  that  used  the  GSS. 

Aligning  the  Measurement  Scales 

The  surveys  given  to  the  two  participant  groups,  face-to-face  and  GSS  supported 
were  slightly  different  in  nature.  The  non-GSS  group  scale  had  an  additional  question 
asking  if  the  participant  would  use  an  electronic  meeting  system  for  strategic  planning  if 
they  had  the  opportunity.  Eliminating  this  question  allowed  the  two  scales  to  be 
evaluated  for  measurement  factors  as  one  26  item  scale. 

Evaluating  the  Experts 

With  such  a  low  number  of  experts  (n  =  7),  great  care  should  be  taken  to  ensure 
no  one  expert  can  sway  the  evaluation  of  a  squadron’s  action  plan  significantly.  This 
expert  bias  may  be  common  in  this  instance.  Since  the  action  plans  are  produced  from 
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such  a  diverse  selection  of  squadrons,  evaluating  them  becomes  a  difficult  chore,  as  each 
must  be  looked  at  from  the  point  of  view  of  the  operating  squadron.  An  expert  in 
logistics,  for  insteince,  may  judge  other  logistic  squadron  action  plans  harsher  since  the 
expert  is  knowledgeable  not  only  in  strategic  planning,  but  the  specific  task  domain  for 
such  a  squadron  as  well.  Without  knowing  the  complete  histories  of  each  expert,  it  is 
nearly  impossible  to  determine  if  such  a  bias  occurred.  An  analysis  of  the  number  of 
times  an  expert  chose  a  particular  rating,  though,  will  show  if  they  were  generally  hard  or 
easy  raters.  This  may  lead  to  the  elimination  of  an  expert  seen  to  be  “overly  harsh”  or 
“overly  easy.”  The  number  of  times  each  expert  chose  each  of  the  seven  possible  ratings 
is  shown  in  Table  1 . 


Table  1.  Expert  Rating  Selection  Frequencies 


Rating 

Values 

1 

2 

3 

4 

5 

6 

7 

Expert  1 

91 

18 

7 

4 

0 

0 

0 

Expert  2 

0 

5 

19 

18 

54 

24 

0 

Expert  3 

19 

13 

26 

19 

23 

18 

2 

Expert  4 

0 

2 

7 

21 

42 

36 

12 

Expert  5 

17 

4 

11 

46 

38 

3 

1 

Expert  6 

5 

20 

23 

28 

27 

14 

3 

Expert  7 

1 

7 

17 

17 

38 

32 

8 

As  can  be  clearly  seen,  there  is  sufficient  evidence  to  eliminate  Expert  1  as  a 
biased  evaluator  of  the  action  plans.  Of  the  possible  120  ratings  given  by  this  expert,  the 
lowest  rating,  a  value  of  one,  was  chosen  91  times,  or  over  75.8  percent  of  the  time.  No 
other  expert  used  any  one  rating  more  than  45  percent  of  the  time  (Expert  2  with  rating 
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value  5).  Thus,  Expert  1  was  considered  an  “outlier”  and  his  ratings  were  eliminated 
from  any  further  analysis. 

The  elimination  of  the  one  expert  shifted  the  reviewer  demographics.  The  panel 
now  consisted  of  six  Quality  Improvement  officers  with  a  mean  of  20.8  years  of  service 
in  the  Air  Force  and  a  mean  of  3.16  years  working  on  strategic  planning. 

Scale  Item  Correlations 

Scale  correlation  matrices  show  the  degree  to  which  the  questions  on  the  scale 
correlate  with  one  another.  As  such,  it  is  essentially  a  measure  of  the  reliability  of  the 
instrument  to  accurately  measure  a  construct  (Cronbach,  1971).  For  a  scale  that  measures 
only  one  construct,  one  would  expect  the  questions  to  be  highly  correlated,  that  is  if  a 
respondent  selected  a  high  rating  on  question  one,  then  the  same  respondent  would  be 
expected  to  select  a  high  rating  on  all  of  the  other  questions.  The  expert  scale  in  this 
study  shows  this  very  phenomenon.  The  scale  correlation  matrix  for  the  expert  rating 
scale  is  shown  in  Table  2. 


Table  2.  Expert  Scale  Correlation  Matrix: 


Ql 

Q2 

Q3 

Q4 

Q5 

Q6 

Ql 

1.0000 

Q2 

0.2112 

1.0000 

Q3 

0.1275 

0.4777 

1.0000 

Q4 

0.1961 

0.3094 

0.4754 

1.0000 

Q5 

0.2048 

0.2851 

0.3325 

0.3855 

1.0000 

Q6 

0.3611 

0.3855 

0.4574 

0.5183 

0.5040 

1.0000 

For  a  scale  that  measures  more  than  one  factor,  one  would  expect  to  find  groups 


of  questions  that  are  highly  correlated  with  each  other.  If  a  participant  selected  a  high 
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rating  on  one  question  measuring  that  construct,  then  the  same  participant  would  be 
expected  to  select  a  high  rating  on  other  questions  measuring  that  same  factor.  The 
participant  scale  appears  to  be  an  example  of  this  event.  The  scale  correlation  matrix  for 
the  participant  rating  scale  is  shown  in  Table  3. 
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Table  3.  Participant  Scale  Correlation  Matrix 
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Factor  Structure  within  the  Expert  Measurement  Scale 


Principle  axis  factoring  was  used  to  extract  an  initial  set  of  potential  factors  from 
the  obtained  data  for  the  expert  reviewer  surveys.  The  six  questions  were  designated  as 
variables  Q1-Q6,  with  the  first  question  being  assigned  the  variable  name  Ql,  the  second, 
Q2,  and  so  on..  The  calculations  extracted  only  one  factor  with  the  loadings  shown  in  the 
unrotated  factor  matrix  given  in  Table  4. 


Table  4.  Unrotated  Item-Factor  Loadings  for  Expert  Reviewers  Quality  Survey 


Factor  1 

Ql 

.44702 

Q2 

.65159 

Q4 

.73071 

Q5 

.67237 

Q6 

.81262 

Here,  factor  1  accoimted  for  over  46  percent  of  the  total  response  variance  with  no 
other  factors  being  extracted.  The  resultant  scree  plot  for  the  expert  reviewers  quality 
scale  is  shown  in  figure  1 . 

The  pronounced  break  in  the  scree  plot  between  factor  1  and  the  other  factors,  and 
the  observation  that  all  survey  items  loaded  strongly  on  that  factor  suggested  this  scale 
was  indeed  unidimensional.  The  factor  reported  to  be  measured  previously  was  the 
quality  of  the  action  plan  and  is  referenced  as  such  here. 
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Facta  Scree  Rot 


F^crMnter 

Figure  1.  Factor  Scree  Plot  for  Expert  Reviewer  Quality  Survey 

Cautionary  Note  on  the  Results  of  the  Expert  Factor  Analysis 

Although  the  factor  loadings  for  the  expert  reviewer’s  quality  survey  appear 
strong,  the  reader  is  advised  to  be  very  critical  of  those  results.  While  the  n  was  756,  the 
number  of  questions  was  only  six,  with  each  of  the  six  experts  using  the  six-item  survey 
to  evaluate  the  21  different  squadron’s  action  plans.  It  is  entirely  possible  that  the  factor 
analysis  efforts  were  not  given  enough  items  to  find  an  imderlying  factor  structure  within 
the  context  of  this  particular  measurement  instrument.  Factor  analysis  itself  could  try  to 
extract  the  underlying  facets  of  a  multi-faceted  construct  such  as  quality  with  the 
assumption  that  there  are  multiple  items  associated  with  each  of  these  facets.  If  such 
facets  actually  exist  in  this  measurement  instrument,  they  may  have  had  only  one  or  two 
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associated  survey  items.  Thus,  the  apparent  scale  unidimensionality,  suggested  by  the 
item-factor  loadings  and  scree  plot  reported  previously  for  the  expert  reviewer’s  quality 
survey,  could  be  accounted  for  by  highly  intercorrelated  sub-scales  of  quality — ^subscales 
which  simply  did  not  have  enough  individual  items  to  produce  separate  and  discernible 
factors.  This  does  not,  however,  invalidate  the  analysis  or  conclusions  to  follow. 

Quality,  even  in  its  most  aggregated  form  is  a  valid  measure  for  the  action  plans 
evaluated.  There  just  may  be  some  facet  of  quality  that  overwhelms  the  rest  of  the  facets. 
This  overwhelming  facet  may  influence  an  expert  to  provide  highly  correlated  measures 
across  the  scale.  It  is  not  possible  to  discern  which  facet  influences  what  expert  from  the 
data  provided. 

Factor  Structure  within  the  Participant  Measurement  Scale 

The  same  procedure  as  discussed  above  for  the  expert  reviewer’s  scale  was  used 
to  examine  the  quality  scale  administered  to  the  strategic  planning  participants.  A  factor 
matrix  for  the  quality  survey  administered  to  all  strategic  planning  participants,  both 
those  that  were  face-to-face  and  those  that  were  supported  by  the  GSS  is  presented  in 
Table  5. 


Table  5.  Unrotated  Item-Factor  Loadings  for  Participant  Quality  Survey 


Factor  1 

Factor  2 

Ql 

0.6223 

0.0391 

Q2 

0.8014 

-0.1842 

Q3 

0.0639 

0.2231 

Q4 

0.8386 

0.1109 

Q5 

0.8283 

-0.0027 

Q6 

0.5271 

0.5355 

Factor  3  Factor  4 


-0.0393 

-0.3651 

0.1712 

0.1275 

0.6173 

0.6068 

0.0343 

-0.1486 

-0.2813 

0.0918 

0.1935 

0.1024 
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Table  5.  Unrotated  Item-Factor  Loadings  for  Participant  Quality  Survey  (Continued) 


Q7 

0.7932 

-0.0313 

Q8 

0.7262 

-0.1637 

Q9 

0.8005 

-0.1513 

-0.0555 

0.1813 

QIO 

0.7578 

-0.2804 

0.1538 

0.0936 

Qll 

0.7264 

-0.0875 

0.2234 

-0.2087 

Q12 

0.6807 

-0.3094 

0.0743 

0.3094 

Q13 

0.6893 

-0.1818 

0.0643 

0.0235 

Q14 

0.7313 

0.1112 

0.3027 

-0.1124 

Q15 

0.7513 

0.0511 

0.3027 

-0.3542 

Q16 

0.7436 

0.0044 

0.2656 

-0.1897 

Q17 

0.6792 

0.0537 

-0.4281 

0.1390 

Q18 

0.2629 

0.7105 

-0.3216 

0.2205 

Q19 

0.8045 

-0.2452 

-0.0694 

0.1860 

Q20 

0.7154 

-0.2110 

-0.0386 

0.3194 

Q21 

0.8323 

-0.0052 

-0.2520 

0.0087 

Q22 

0.8082 

0.0338 

-0.0383 

-0.1630 

Q23 

0.8334 

0.0874 

-0.0457 

-0.1127 

Q24 

0.6616 

0.3657 

0.1511 

-0.0832 

Q25 

0.8363 

-0.0252 

-0.2650 

0.1167 

Q26 

0.6012 

0.4086 

-0.0379 

0.0455 

Although  the  four-factor  structure  indicated  above  appears  somewhat  chaotic, 
Factor  1  actually  accounted  for  over  51  percent  of  the  total  variance  in  item  responses, 
with  the  next  highest  factor.  Factor  2,  contributing  only  another  6  percent,  a  ratio  of  better 
than  8:1  compared  to  the  first  two  eigen  values.  The  fact  that  all  items  had  sizeable  loads 
on  the  first  factor,  coupled  with  a  pronounced  break  in  the  scree  plot  between  factor  1  and 
the  remaining  factors  (see  Figure  2),  suggested  that  the  participant  quality  scale  was 
unidimensional. 
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Factor  Scree  Rot 


Factor  Number 

Figure  2.  Factor  Scree  Plot  for  Participant  Quality  Survey 

Since  the  previous  work  reported  only  one  scale  reliability,  but  that  two 
constructs,  satisfaction  and  commitment,  were  measured,  the  single  factor  unrotated 
solution  was  not  expected.  A  close  examination  of  the  survey  questions  lends  support  to 
a  one  factor  scale  measuring  overall  quality.  There  are  questions  covering  the 
participant’s  satisfaction  with  the  action  plans  produced  as  well  as  questions  assessing 
their  satisfaction  with  the  procedures  used.  Thus,  the  apparent  scale  unidimensionality, 
suggested  by  the  item-factor  loadings  and  scree  plot  reported  previously  for  the 
participant’s  survey,  could  be  accounted  for  by  highly  intercorrelated  sub-scales  of 
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quality — subscales  which  simply  did  not  have  enough  individual  items  to  produce 
separate  and  discernible  factors  with  an  unrotated  factor  analysis. 

Further  evaluation  of  the  participant  scale  was  then  done  using  a  varimax  rotation 
to  see  if  the  expected  factors  emerged.  The  varimax  procedure  converged  in  only  eight 
iterations  producing  the  results  of  the  analysis  shown  in  Table  6. 

This  analysis  clearly  produces  the  expected  two  factor  solution.  Factor  1  is 
represented  by  12  items,  while  factor  2  is  represented  with  10  items.  The  survey 
questions  themselves  seem  to  define  the  first  factor  as  the  satisfaction  with  product,  while 
the  second  factor  assesses  the  participant’s  satisfaction  with  the  procedure  used  to 
develop  the  product,  in  this  case,  the  strategic  action  plans. 


Table  6.  Rotated  Item-Factor  Loadings  for  Participant  Quality  Survey 


Factor  1 

Factor  2 

Factor  3 

Factor  4 

Ql 

0.2745 

0.5967 

0.1590 

-0.2589 

Q2 

0.6704 

0.4897 

0.0455 

0.1745 

Q3 

0.0476 

0.0025 

0.1129 

0.8878 

Q4 

0.4726 

0.6484 

0.3031 

-0.0552 

Q5 

0.7100 

0.3331 

0.3651 

-0.1594 

Q6 

0.1214 

0.4308 

0.5693 

0.2970 

Q7 

0.5444 

0.5650 

0.1539 

-0.1349 

Q8 

0.5926 

.  0.3630 

0.2418 

-0.1228 

Q9 

0.7390 

0.3523 

0.1661 

0.0431 

QIO 

0.6736 

0.4621 

-0.0510 

0.1241 

Qll 

0.4076 

0.6797 

0.0187 

0.0169 

Q12 

0.7496 

0.2431 

-0.0262 

0.1968 

Q13 

0.5712 

0.4299 

0.0313 

0.0288 

Q14 

0.3402 

0.6880 

0.1822 

0.1709 

Q15 

0.2746 

0.8373 

0.0870 

0.0071 

Q16 

0.3743 

0.7108 

0.0915 

0.0759 

Q17 

0.6373 

0.1473 

0.4315 

-0.2298 

Q18 

0.0411 

-0.0322 

0.8503 

0.0143 

Q19 

0.7915 

0.3338 

0.0925 

0.0200 
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Table  6.  Rotated  Item-Factor  Loadings  for  Participant  Quality  Survey  (Continued) 


Q20 

0.7631 

0.2170 

0.1114 

Q21 

0.6690 

0.3989 

0.3366 

Q22 

0.4998 

0.5934 

0.2509 

-0.1317 

Q23 

0.5173 

0.5824 

0.3174 

Q24 

0.2252 

0.5851 

0.4407 

Q25 

0.7330 

0.3284 

0.3473 

Q26 

0.2700 

0.3848 

0.5539 

Scale  Reliability  Measures 

Next,  the  scales  extracted  during  the  factor  analysis  were  evaluated  for  reliability 
using  Cronbach’s  alpha.  Since  there  were  two  factors  extracted  from  the  participant 
survey  instrument,  two  different  calculations  were  made.  The  results  of  the  analysis  for 
the  first  factor  gave  a  scale  alpha  of  .94  for  the  12  item  scale  measuring  Factor  1 .  The 
results  are  presented  in  Table  7. 


Table  7.  Reliability  Analysis  for  Participant  Factor  1 


Scale  Mean  if  Item 

Scale  Variance  if 

Corrected  Item-Total 

Alpha  if  Item 

Deleted 

Item  Deleted 

Correlation 

Deleted 

Q2 

57,4289 

170.6834 

0.7739 

0.9388 

Q5 

57.  0448 

167.5324 

0.8192 

0.  9372 

Q8 

57.  3874 

166.2634 

0.  6900 

0.  9424 

Q9 

57.  3383 

169.9572 

0.  7959 

0.  9381 

QIO 

57.  2777 

167.3450 

0.7451 

0.  9398 

Q12 

57.  7997 

171.9423 

0.  6766 

0.  9420 

Q13 

57. 6851 

169.6386 

0.  6576 

0.  9432 

Q17 

57.  3510 

174.6599 

0.  6688 

0.  9422 

Q19 

57.  5019 

170.7023 

0.  8047 

0.  9380 

Q20 

57.  5989 

175.5525 

0.7163 

0.  9409 

Q21 

57.  1596 

164.5234 

0.  8042 

0.  9376 

Q25 

57.  0046 

168.1901 

0.  8287 

0.  9370 

With  no  factors  having  a  significantly  higher  alpha  if  the  item  was  deleted,  all 


items  were  kept  for  subsequent  analysis.  Factor  2  analysis  was  completed  the  same  as  for 


38 


Factor  1 .  The  resultant  alpha  for  the  10  item  scale  measuring  Factor  2  was  .93  and  is 
shown  in  Table  8. 


Table  8.  Reliability  Analysis  for  Participant  Factor  2 


Scale  Mean  if  Item 

Scale  Variance  if 

Corrected  Item- 

Alpha  if  Item 

Deleted 

Item  Deleted 

Total  Correlation 

Deleted 

Qi 

48.4116 

125.7225 

0.6077 

0.9309 

Q4 

47.7260 

122.1091 

0.8049 

0.9216 

Q7 

48.1237 

121.4800 

0.7338 

0.9247 

Qll 

47.8862 

121.6310 

0.7224 

0.9252 

Q14 

47.3877 

121.3317 

0.7142 

0.9257 

Q15 

47.5383 

115.9027 

0.7902 

0.9218 

Q16 

47.5753 

120.0777 

0.7368 

0.9245 

Q22 

47.7388 

123.2780 

0.7864 

0.9226 

Q23 

47.6884 

119.2335 

0.7959 

0.9214 

Q24 

47.7897 

124.1651 

0.6452 

0.9291 

Once  again,  with  no  factors  having  a  significantly  higher  alpha  if  the  item  was 
deleted,  all  items  were  kept  for  subsequent  analysis. 

The  analysis  for  the  Expert  Evaluator’s  Quality  Scale  was  completed  in  the  same 
manner  with  the  overall  scale  alpha  being  a  much  lower  .7601,  but  still  above  .70  which 
is  generally  considered  acceptable  for  behavioral  research  (Nunnally,  1967),  as  shown  in 
Table  9.  Thus,  this  scale  has  a  high  reliability  which  means  the  evaluation  of  the 
accuracy  of  the  instrument  to  measure  the  one  extracted  factor  is  very  favorable.  One 
would  expect  this  instrument  to  provide  reliable  measmes  of  the  factor  as  a  function  of 
the  respondents  understanding  of  the  question  and  not  as  a  result  of  a  variation  in  true 
score  (Cronbach,  1971). 


39 


Table  9.  Reliability  Analysis  for  Expert  Evaluator  Factor  1 


Scale  Mean  if  Item 

Scale  Variance  if  Item 

Corrected  Item-Total 

Alpha  if  Item 

Deleted 

Deleted 

Correlation 

Deleted 

Qi 

21.2067 

25.9514 

0.3091 

0.7675 

Q2 

22.2241 

20.4878 

0.4723 

0.7408 

Q3 

20.6845 

23.3853 

0.5628 

0.7152 

Q4 

21.0257 

21.9927 

0.5471 

0.7132 

Q5 

21.9210 

21.9392 

0.4949 

0.7271 

06 

21.9555 

20.0234 

0.6647 

0.6778 

Once  again,  with  no  factors  having  a  significantly  higher  alpha  if  the  item  was 
deleted,  all  items  were  kept  for  subsequent  analysis. 

Assessing  the  Differences  in  Action  Plan  Quality 

Given  the  validity  and  reliability  of  the  scales  used  in  this  study,  further  analysis 
examined  the  resulting  differences  in  scale  scores  between  the  groups.  The  reviewer’s 
scoring  was  matched  to  the  appropriate  experimental  group,  face-to-face  and  GSS.  When 
this  was  done,  a  standard  paired  t-test  was  used  to  determine  if  there  were  differences  in 
the  experts’  evaluations  of  the  action  plans.  For  each  expert  evaluator,  their  scores  for  all 
GSS  supported  squadron  action  plans  were  totaled  and  then  averaged  to  provide  a 
composite  score.  A  similar  composite  score  was  calculated  for  their  Face-to-Face 
composite  scores.  These  scores  were  then  paired  and  the  t-test  performed.  The  results 
reveal  that  there  was  indeed  a  difference  between  sample  means  (t  =  2.96,  p  <  .05),  and 
more  specifically  that  the  mean  total  score  of  the  GSS  supported  group  (pcss  =  4.63)  was 
significantly  higher  than  the  mean  total  score  of  the  face-to-face  group  (Pnr  4.14)  that 
did  not  use  GSS.  This  suggests  that  the  GSS  supported  groups  produced  significantly 
higher  quality  action  plans  than  the  traditional  face-to-face  groups. 
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Assessing  the  Differences  in  Participant  Perceptions 


Two  sample  t-tests  were  used  to  determine  the  differences  in  satisfaction  with 
product,  satisfaction  with  process,  and  overall  satisfaction  between  the  two  groups. 
Satisfaction  with  the  strategic  planning  process  was  higher  in  the  GSS  supported  groups 
(M'Gss  “  5.63)  than  the  non-supported  groups  (pnT  “  4.67,  t  =  4.12,  p  <  .05).  The  GSS 
supported  groups  also  reported  higher  satisfaction  with  the  action  plans  produced  (t  = 
2.07,  p  <  .05,  Pgss  =  5.39  >Pftf'=  4.87).  The  t-test  for  the  overall  satisfaction  for  a  one 
factor  solution  also  favored  the  GSS  supported  groups  (t  =  2.92,  p  <  .05,  poss  ~  5.50  > 

=  4.78).  All  of  these  results  support  the  idea  of  GSS  use  increasing  the  participants’ 
self-reported  satisfaction  with  both  the  process  used  to  create  the  action  plans  and  the 
action  plans  themselves. 

Further  Participant  Results 

As  stated  previously,  the  GSS  supported  groups  used  a  process  that  limited  their 
action  plan  generation  time  to  one  meeting  that  took  less  than  8  hours  for  the  entire 
production  of  the  action  plans.  The  Non-supported  group  participants  reported  their 
approximate  time  spent  and  number  of  meetings  in  their  survey.  Of  the  42  respondents, 
the  average  number  of  meetings  was  3.4,  ranging  as  high  as  8  meetings,  to  complete  the 
action  plan  development.  The  average  time  spent  on  the  strategic  planning  process  for 
the  Non-GSS  groups  was  over  17.7  hours,  varying  from  90  hours  to  a  low  of  five  hours 
while,  once  again,  the  GSS  groups  met  once  for  an  average  of  less  than  8  hours. 
Additionally,  while  all  GSS  groups  finished  their  action  plans  in  the  starting  month  of 
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August,  the  Non-GSS  groups  finished  as  late  as  March,  with  the  average  finish  in 
December. 

Post  Hoc  Analysis 

To  determine  if  the  differences  in  satisfaction  between  the  various  groups  had  any 
predictive  abilities  as  to  the  quality  of  the  product  as  evaluated  by  the  experts,  stepwise 
linear  regressions  were  performed  for  each  group  and  for  the  overall  population.  The  first 
regression  included  all  participants,  both  GSS  and  face-to-face  groups  satisfaction  ratings 
with  the  paired  reviewers’  evaluation  used  as  the  dependent  variable.  The  resulting 
model  had  no  significant  variables  except  the  constant,  which  is  not  meaningful  in  the 
context  of  this  study,  see  Table  10. 

Table  10.  Stepwise  Regression  of  Expert  Rating  for  All  Participants 


Variablefs)  Included 

& 

E! 

MSE 

Constant 

25.75 

0.00 

24.25 

VariableCs')  Excluded 

I 

p 

Satisfaction  with  Process 

-1.06 

0.30 

Satisfaction  with  Product 

-0.85 

0.41 

Stepwise  at  .10 


The  next  series  of  stepwise  linear  regressions  separated  out  the  groups  again  and 
paired  them  with  their  expert  ratings.  The  model  for  the  face-to-face  groups  eliminated 
both  of  the  satisfaction  variables,  leaving  only  the  constant  once  again,  see  Table  11. 
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Table  1 1 .  Stepwise  Regression  of  Expert  Rating  for  FtF  Participants 


Variablefs)  Included 

& 

MSE 

Constant 

24.73 

0.00 

19.32 

Variablefs)  Excluded 

I 

P 

Satisfaction  with  Process 

-1.64 

0.13 

Satisfaction  with  Product 

-1.22 

0.24 

Stepwise  at  .10 


The  model  for  the  GSS  group  eliminated  the  satisfaction  with  product  variable, 
but  kept  the  satisfaction  with  process  variable  since  it  served  as  a  significant  predictor  of 
the  expert  evaluation  (P  =  -.89,  p  <  .10).  It  should  be  noted  that  while  satisfaction  with 
process  was  not  eliminated  for  this  model,  it  provided  only  another  .05  to  the  overall  R^ 
of  .56.  As  such,  this  factor,  though  significant,  adds  little  to  the  overall  predictive  ability 
of  the  model.  The  results  of  this  analysis  is  presented  in  Tablel2. 


Table  12.  Stepwise  Regression  of  Expert  Rating  for  GSS  Participants 


Variablefs)  Included 

& 

MSE 

Constant 

78.38 

0.56 

16.83 

Satisfaction  with  Process 

-0.89 

Variables')  Excluded 

I 

P 

Satisfaction  with  Product 

-0.85 

0.41 

Stepwise  at  .10 


Synthesis 

After  reporting  the  original  results  from  the  previous  statistical  analysis,  this 
chapter  started  with  a  rigorous  examination  of  the  experts  which  led  to  the  elimination  of 
one  expert  due  to  extreme  bias.  This  was  followed  by  a  series  of  validation  and  reliability 
tests  of  the  measurement  instrument,  namely  the  two  post  process  surveys  administered  to 
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the  participants  and  the  evaluation  scale  used  by  the  experts  to  rate  the  action  plans.  This 
led  to  the  identification  of  two  satisfaction  measures  for  the  participant  scales  and  an 
overall  quality  measure  from  the  expert  instrument,  each  with  a  high  reliability  measure. 
Only  these  significant  factors  were  used  in  the  resulting  analysis.  Next,  the  validated 
factors  were  tested  to  discover  differences  in  the  qxiality  of  the  action  plans  produced  by 
the  two  groups  as  rated  by  the  panel  of  experts.  Then,  the  participants’  self  reported 
measures  of  satisfaction  with  process  and  satisfaction  with  product  were  tested.  In  all 
cases,  both  the  existence  and  the  direction  of  these  differences  were  then  reported. 
Finally,  these  variables  were  used  in  stepwise  linear  regressions  to  test  their  ability  to 
predict  the  quality  of  the  action  plans  produced. 
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V.  Results 


Discussion 

Strategic  planning  is  performed  throughout  the  U.S.  Air  Force  as  a  leadership 
decision  making  process  to  determine  the  focus  and  direction  of  organizations  at  all 
levels.  In  today’s  downsizing  force,  the  strategic  planning  process  is  a  vital  element  in 
the  distribution  and  allocation  of  scarce  resources.  The  current  process  is  extremely  time 
consuming,  taking  three  to  four  months  to  complete  at  most  wings.  A  computer-mediated 
strategic  planning  process  is  supposed  to  help  reduce  the  constraints  associated  with 
bringing  a  large  group  of  people  together  to  collaborate.  Group  support  systems  are 
technology  designed  to  directly  impact  and  change  the  behavior  of  groups  to  improve 
group  effectiveness,  efficiency  and  satisfaction.  The  data  analyzed  in  the  previous 
section  supports  this  assertion. 

Interpretation  of  Results 

Since  the  propositions  were  developed  from  the  literature  review  and 
further  refine  the  stated  research  focus  to  evaluate  the  efforts  of  the  Air  Force  research 
project  created  to  improve  strategic  planning  at  Air  Force  wing  level  using  a  computer- 
mediated  communication  group  support  system,  they  will  be  used  to  outline  the 
interpretation  of  results. 

Proposition  1:  Higher  quality  action  plans  will  be  produced  by  groups  employing 
GSS  than  by  traditional  face-to-face  groups. 

This  proposition  was  supported  by  the  data  analysis.  The  expert  panels  scores 
showed  a  statistically  significant  higher  mean  score  for  the  GSS  supported  groups.  While, 
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generally  speaking,  the  review  of  recent  research  predicted  this  would  happen,  the  degree 
to  which  the  groups  would  diverge  was  not  predictable.  In  this  case  the  effect  was  quite 
small,  moving  the  face-to-face  average  of  4.14,  roughly  neutral,  to  only  4.63,  or  about 
half  a  rating  higher,  for  the  GSS  groups.  That  both  groups  mean  score  was  near  neutral 
as  judged  by  the  experts  is  somewhat  disconcerting  though.  This  may  be  an  affect  of  the 
scoring,  the  lack  of  experience  in  the  participant  members  with  strategic  planning,  or  an 
indication  that  the  expert  reviewers  diverged  in  the  overall  opinions  for  each  action  plan 
reviewed.  While  the  means  were  statistically  significant  in  identifying  differences 
between  GSS  and  Non  GSS,  these  differences  did  not  apparently  have  a  “dramatic” 
influence  of  quality.  Basically,  there  were  differences  between  the  products  produced 
using  the  GSS  and  those  created  using  the  more  traditional,  Face-To-Face  meeting 
method,  but  it  is  open  to  debate  how  practically  important  these  differences  were.  This  is 
not  to  say  that  the  differences  should  be  dismissed,  though,  but  rather  they  are  worthy  of 
further  examination  because,  while  the  effect  is  positive,  the  potential  value  is  still 
uncertain.  Overall  though,  this  proposition  lends  support  to  the  continued  adoption  of 
GSS  technology  for  these  types  of  problems. 

Proposition  2:  Groups  employing  GSS  will  report  a  higher  satisfaction  with  the 
decision  process  than  traditional  face-to-face  groups. 

This  proposition  was  also  supported  by  the  data  analysis.  The  participants’  self 
reported  satisfaction  with  process  scores  showed  a  higher  rating  for  the  GSS  supported 
groups.  This  means  that  the  members  who  used  the  GSS  to  produce  their  action  plans 
were  more  satisfied  with  the  process  used  to  develop  their  action  plans  than  the  members 
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of  the  traditional  face-to-face  groups.  The  effect  was  relatively  larger  for  this  effect  than 
the  other  survey  measured  factors.  The  mean  was  only  4.67  for  the  face-to-face  group, 
but  the  GSS  groups  average  reported  scores  were  nearly  a  whole  rating  higher  at  5.63. 
This  is  a  clear  indication  of  group  satisfaction  with  the  process  used  to  develop  action 
plans  increasing  due  to  GSS  usage.  Therefore,  this  proposition  also  lends  support  to  the 
continued  adoption  of  GSS  technology  for  these  types  of  problems. 

Proposition  3:  Groups  employing  GSS  will  report  a  higher  satisfaction  with  the 
action  plan  produced  than  traditional  face-to-face  groups. 

This  proposition  was  also  supported  by  the  data  analysis.  The  participants’  self 
reported  satisfaction  with  product  ratings  also  showed  a  higher  mean  for  the  GSS 
supported  groups.  This  effect  by  the  mean  differing  from  4.87  for  the  face-to-face  groups 
upward  to  5.63  for  the  GSS  groups.  This  is  a  clear  indication  of  group  satisfaction  with 
the  product,  in  this  case  the  action  plans,  being  rated  significantly  higher  with  GSS  usage. 
Therefore,  this  proposition  also  lends  support  to  the  continued  adoption  of  GSS 
technology  for  these  types  of  problems. 

Proposition  4:  Groups  employing  GSS  will  complete  the  action  plan  development 
process  faster  than  traditional  face-to-face  groups. 

This  proposition  was  the  most  strongly  supported  of  all  the  propositions  by  the 
data  analysis.  The  use  of  the  GSS  cut  the  number  of  meetings  from  an  average  of  3.4 
down  to  just  one.  Due  to  the  delays  in  scheduling  and  just  the  logistical  details  of 
gathering  everyone  together,  the  GSS  provided  greater  efficiency,  since  some  squadrons 
did  not  finish  the  process  until  into  the  next  calendar  year,  even  though  all  squadrons  had 
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an  August  start  date.  Even  without  the  enhancement  in  effectiveness  and  satisfaction  that 
a  GSS  can  provide,  this  result  of  greater  efficiency  alone  would  lend  support  to  efforts  to 
further  the  spread  and  use  of  GSS  technology  for  these  sorts  of  decision  making 
processes. 

Linear  Regressions.  While  overall,  the  results  of  the  linear  regressions  were 
mildly  disappointing,  there  was  some  indication  of  a  rather  surprising  result. 

Specifically,  the  result  of  the  GSS  linear  regression  that  had  satisfaction  with  process  as 
the  only  significant  variable  left  in  the  model,  eliminating  satisfaction  with  product  as  a 
significant  predictor  of  the  experts’  evaluation  of  the  action  plans.  This  indicates  that  the 
participants’  self-reported  satisfaction  with  the  process  used  to  create  the  action  plans  was 
a  stronger  indicator  of  the  quality  of  the  action  plan,  as  gauged  by  the  expert  panel,  than 
their  own  assessment  of  the  same  plan.  Such  reasoning  implies  there  is  something 
qualitatively  better  about  GSS  use  that  leads  to  a  quantitatively  better  product.  Such  a 
result  calls  for  further  evaluation  in  future  efforts  before  any  definite  statements  can  be 
made,  but  it  is  still  a  compelling  result  from  this  analysis. 

Implications 

For  strategic  planners  and  information  technology  professionals  in  the  Air  Force, 
the  results  of  these  analyses  are  compelling.  Both  groups  should  be  actively  learning 
more  about  this  technology  and  its  potential  benefits  to  them  and  their  peers.  If  such 
results  become  the  norm  from  Air  Force  field  studies,  the  prevalence  of  these  type  of 
computer  mediated  communication  systems  should  increase. 
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Researchers  should  find  the  extensive  examination  and  validation  of  the 
measurement  instruments,  especially  the  participant  scales,  most  beneficial.  Future  use  of 
these  scales  will  preclude  the  normal  scale  development  exercises  common  to  research 
efforts.  With  these  validated  scales,  the  researcher  can  spend  more  time  on  experimental 
design  and  data  gathering  knowing  that  the  constructs  of  satisfaction  with  process  and 
satisfaction  with  product  are  being  adequately  and  reliably  measured. 

Limitations 

The  original  data  gathering  did  have  some  basic  limitations.  Foremost  is  probably 
the  problems  with  the  expert  reviewers.  First,  there  were  a  limited  number  of  experts 
selected  to  do  the  reviewing.  With  a  larger  group  of  experts,  the  variations  seen  in  the 
scores  and  the  low  inter-item  correlations  may  have  been  avoided.  Second,  the 
experience  level  of  the  selected  experts  in  this  sample  was  less  than  might  have  been 
expected.  Strategic  planning  is  a  complex  process,  and  three  years  experience  may  not  be 
enough  experience  to  be  considered  an  “expert”  in  this  arena.  Finally,  the  scale  used  to 
gauge  the  quality  of  the  action  plans  may  be  insufficient  because,  effectively,  the  one 
factor  extracted  may  be  an  artifice  of  the  factor  analysis  method.  Given  only  six  items, 
there  was  little  that  could  be  done  using  factor  analysis  to  try  and  separate  the  various 
types  or  subscales  of  quality  that  should  be  measured  when  gauging  an  action  plan  output 
from  the  strategic  planning  process.  The  reviewers  score,  then,  is  at  best  a  general  signal 
from  somewhat  experienced  raters  as  to  the  overall  quality  of  the  submitted  action  plans 
without  anything  concrete  to  be  said  about  any  specific  characteristic  for  any  particular 
plan  reviewed. 
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Suggestions  for  Future  Research 

Future  researchers  could  remedy  all  of  the  above  limitations  by  modifying  and 
validating  existing  quality  measures.  These  scales  would  enable  the  researcher  to 
determine  which  of  the  characteristics  of  quality  are  most  mediated  through  the  use  of  the 
GSS.  Further  research  should  also  focus  on  the  real  variable  of  interest  in  this  type  of 
research — ^performance.  As  these  organizations  start  their  annual  process  with  the  review 
of  last  year’s  action  plans,  it  would  be  beneficial  to  get  their  own  ratings  as  to  the 
previous  year’s  action  plan  quality,  or  effectiveness.  A  measure  of  the  perception  of  how 
last  year’s  action  plan  helped  them  to  focus  their  efforts  and  perform  would  also  be  useful 
information  to  gather.  Granted,  the  turnover  in  personnel  from  last  year  will  be 
problematic,  but  not  so  devastating  as  to  invalidate  the  effort.  These  efforts,  along  with  a 
repetition  of  the  previous  work  would  naturally  lend  itself  to  a  longitudinal  study  of  the 
GSS  effects  on  the  strategic  formation  process  adding  validity  to  any  results  obtained. 
Finally,  efforts  could  be  made  to  determine  the  underlying  reasons  for  the  favorable 
effects  reported  here.  Varying  the  task  type  or  environment  may  lend  some  insight  into 
this  question. 

Conclusion 

It  is  clear  that  there  are  still  questions  about  how  to  properly  conduct  strategic 
planning  to  produce  the  highest  quality  plans  in  the  most  effective  and  efficient  manner. 
To  answer  this  question,  the  Air  Force  has  created  a  research  project  aimed  at  discovering 
new  methods  to  improve  strategic  planning  at  Air  Force  wing  levels  using  a  computer- 
mediated  communication  system  as  a  tool.  The  research  project’s  goal  is  to  apply 
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collaborative  processes  and  technologies,  such  as  GSSs,  to  improve  the  quality  of  the 
strategic  planning  process  for  the  Air  Force.  This  thesis  analyzed  the  data  from  one  of 
those  studies.  The  thorough  evaluation  of  the  measurement  instruments  led  to  validated 
scales  that  could  be  valuable  for  future  use.  Using  this  scale,  the  results  report  finding  a 
significant  increase  in  overall  plan  quality,  and  increased  participant  satisfaction  with 
both  the  process  employed  and  the  product  created  when  a  GSS  was  used.  These  results 
lend  support  to  the  continued  fielding  of  such  systems  to  support  the  decision  making 
process  for  these  types  of  problems.  Furthermore,  continued  research  should  be 
conducted  to  try  and  discover  the  vmderlying  mechanisms  of  these  results  and  increase  the 
already  favorable  effects. 
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Appendix  A.  Reviewer’s  Evaluation  of  Action  Plans 


The  rating  scale  is  where  4  is  the  middle  position  on  the  scale  and  represents  “undecided” 
or  “neutral.”  Moving  out  from  the  center,  a  3  or  5  represents  “slight”  agreement  or 
disagreement,  a  2  or  6  represents  “moderate”  agreement  or  disagreement,  a  1  or  7 
represents  “strong”  agreement  or  disagreement.  Please  answer  honestly;  you  are 
assured  the  information  provided  on  this  questionnaire  is  confidential. 


SA  N 


SD 


7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 


This  squadron’s  action  plans  are  high  quality. 

This  squadron’s  action  plans  are  achievable. 

Implementation  of  these  action  plans  will  be  realistic  for  the 
Squadron. 

These  action  plans  will  receive  a  high  level  of  buy-in  from  the 
squadron. 

This  Squadron’s  action  plans  have  fully  addressed  the  group’s 
targets. 

It  is  clear  how  the  success  of  these  plans  will  be  measured. 


Reviewer  Profile 


How  many  years  have  you  been  in  the  Armed  Forces? _ 

How  many  years  have  you  participated  in  strategic  planning? 
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Appendix  B.  Participant's  Evaluation  of  Action  Plans  Process  (Non  GSS 

supported) 


The  following  questions  concern  the  strategic  planning  process  in  which  you  participated. 
The  rating  scale  is  where  4  is  the  middle  position  on  the  scale  and  represents  “undecided” 
or  “neutral.”  Moving  out  from  the  center,  a  3  or  5  represents  “slight”  agreement  or 
disagreement,  a  2  or  6  represents  “moderate”  agreement  or  disagreement,  a  1  or  7 
represents  “strong”  agreement  or  disagreement.  Please  answer  honestly;  you  are 
assured  the  information  provided  on  this  questionnaire  is  confldential. 


SA  N  SD 


7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 


I  came  up  with  a  lot  of  suggestions  for  the  group  to  consider  during 
our  squadron’s  strategic  planning  process. 

Our  squadron  did  a  good  job  coming  up  with  action  plans. 

We  got  off  onto  a  lot  of  tangents  instead  of  sticking  to  the  issue 
when  we  were  creating  the  action  plans. 

The  procedure  used  to  develop  our  squadron’s  action  plans  was 
effective. 


7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 


I  am  personally  committed  to  implementing  the  action  plans  our 
squadron  developed. 

There  was  LESS  conflict  in  the  interaction  of  our  strategic 
planning  session  than  usual. 

I  was  satisfied  with  my  own  interaction  in  the  strategic  planning  process. 

I  did  NOT  feel  like  I  was  part  of  the  decision  making  process  when 
we  were  developing  action  plans. 

I  think  our  squadron’s  action  plans  are  effective  implementations 
of  our  group’s  targets. 

I  was  disappointed  in  the  quality  of  our  squadron  action  plans. 


7  6  5  4  3  2  1 


The  procedure  our  squadron  used  to  develop  action  plans  was 
difficult  to  follow. 


7  6  5  4  3  2  1 


Our  squadron  developed  the  best  action  plans  possible. 
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7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 


I  do  NOT  feel  much  buy-in  to  the  action  plans  developed  by  our 
squadron. 

The  method  our  squadron  used  to  conduct  the  strategic  planning 
slowed  down  our  group’s  progress. 

Key  issues  did  not  receive  enough  time  or  attention  during  our 
strategic  planning  session. 

The  procedure  our  squadron  used  to  develop  action  plans  inhibited 
our  ability  to  develop  the  best  plans. 

I  am  compelled  to  implement  the  action  plans  our  squadron 
developed. 

There  were  LESS  verbal  challenges  in  the  interaction  of  our 
strategic  planning  session  than  usual. 

I  was  satisfied  with  the  action  plans  our  squadron  developed. 


7  6  5  4  3  2  1 

7  6  5  4  3  2  1 


I  was  happy  with  the  number  of  action  plans  our  squadron  came  up 
with  during  our  strategic  planning  procedure. 

I  felt  free  to  participate  in  the  strategic  planning  process. 


7  6  5  4  3  2  1 


The  procedure  used  to  develop  action  plans  was  easy  for  me  to 
understand. 


7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 


During  the  meeting,  I  was  relaxed  when  we  developed  the  action 
plans. 

The  strategic  planning  process  took  more  time  than  was  productive. 

I  will  work  hard  to  implement  the  action  plans  my  squadron 
developed. 

There  was  MORE  disagreement  in  the  strategic  planning  session 
than  usual. 
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Demographic  Information 


lam  MALE _ FEMALE _ 

How  old  are  you? _ 

How  many  computer-supported  meetings  have  you  participated  in _ ? 

On  average,  how  many  people  met  when  your  squadron  developed  action  plans? 
How  many  times  did  your  squadron  meet  to  discuss  and  complete  action  plans? 
How  many  total  hours  did  YOU  spend  on  this  year’s  strategic  planning  process? 

What  year  and  month  did  your  squadron  complete  its  1997  strategic  plan?  _ 

Which  squadron  are  you  a  member  of _ 1 
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Appendix  C.  Participant’s  Evaluation  of  Action  Plans  Process  (GSS  supported) 


The  following  questions  concern  the  electronic  group  interaction  in  which  you 
participated.  The  rating  scale  is  where  4  is  the  middle  position  on  the  scale  and 
represents  “undecided”  or  “neutral.”  Moving  out  from  the  center,  a  3  or  5  represents 
“slight”  agreement  or  disagreement,  a  2  or  6  represents  “moderate”  agreement  or 
disagreement,  a  1  or  7  represents  “strong”  agreement  or  disagreement.  Please  answer 
honestly;  you  are  assured  the  information  provided  on  this  questionnaire  is 
confidential. 


SA  N  SD 
7  6  5  4  3  2  1 
7  6  5  4  3  2  1 
7  6  5  4  3  2  1 
7  6  5  4  3  2  1 
7  6  5  4  3  2  1 

7  6  5  4  3  2  1 
7  6  5  4  3  2  1 
7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 
7  6  5  4  3  2  1 

7  6  5  4  3  2  1 
7  6  5  4  3  2  1 
7  6  5  4  3  2  1 


I  came  up  with  a  lot  of  suggestions  for  group  to  consider. 

Our  squadron  did  a  good  job  coming  up  with  action  plans. 

We  got  off  onto  a  lot  of  tangents  instead  of  sticking  to  the  issue. 

The  procedure  used  to  develop  these  action  plans  was  effective. 

I  am  personally  committed  to  implementing  the  action  plans  we 
developed. 

There  was  LESS  conflict  in  today’s  interaction  than  usual. 

I  was  satisfied  with  my  own  interaction  in  this  group. 

I  did  NOT  feel  like  I  was  part  of  the  decision  making  process  when 
we  were  developing  action  plans. 

I  think  our  squadrons’  action  plans  are  effective  implementations  of 
the  Group  targets. 

I  was  disappointed  in  the  quality  of  our  squadron  action  plans. 

The  procedure  we  used  to  develop  action  plans  was  difficult  to 
follow. 

Our  squadron  developed  the  best  action  plans  possible. 

I  do  NOT  feel  much  buy-in  to  the  action  plans  we  developed. 

The  use  of  the  electronic  meeting  system  slowed  down  our  groups 
progress. 
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7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 

7  6  5  4  3  2  1 


We  could  have  done  a  better  job  developing  action  plans  if  we  had 
not  used  an  electronic  meeting  systems. 

The  procedure  we  used  to  develop  action  plans  inhibited  our  ability 
to  develop  the  best  plans. 

I  am  compelled  to  implement  the  action  plans. 

There  were  LESS  verbal  challenges  in  the  interaction  than  usual. 

I  was  satisfied  with  the  action  plans  our  group  developed. 

I  was  happy  with  the  number  of  action  plans  we  came  up  with 
today. 

I  felt  free  to  participate  in  this  group. 

The  procedure  used  to  develop  action  plans  was  easy  for  me  to 
understand. 

I  felt  relaxed  using  electronic  meeting  systems  to  develop  action  plans. 

I  would  use  an  electronic  meeting  system  for  strategic  planning  if  I 
had  the  opportunity. 

Using  the  electronic  meeting  system  shorted  the  time  it  took  to 
develop  action  plans. 

I  will  work  hard  to  implement  these  action  plans. 


7  6  5  4  3  2  1 


There  was  MORE  disagreement  in  the  group  than  usual. 


Demographic  Information 


I  am  MALE  FEMALE 


How  old  are  you? _ 

Prior  to  today’s  interaction,  how  many  electronic  meetings  have  you  participated  in? 
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